An Approach to Improving Existing Measurement Frameworks

Software organizations are in need of methods for understanding, structuring, and improving the data they are collecting. This paper discusses an approach for use when a large number of diverse metrics are already being collected by a software organization. The approach combines two methods. One looks at an organization's measurement framework in a top-down fashion and the other looks at it in a bottom-up fashion. The top-down method, based on the goal-question-metric (GQM) paradigm, is used to identify the measurement goals of data users. These goals are then mapped to the metrics being used by the organization, allowing us to: (1) identify which metrics are and are not useful to the organization, and (2) determine whether the goals of data user groups can be satisfied by the data that are being collected by the organization. The bottom-up method is based on a data mining technique called attribute focusing (AF). Our method uses this technique to identify useful information in the data that the data users were not aware of. We describe our experience in analyzing data from a software customer satisfaction survey at ISM to illustrate how the AF technique can be combined with the GQM paradigm to improve measurement and data use inside software organizations. T here are many different groups involved in the processes of developing, maintaining, and managing software. Those groups need to use measurement to characterize, control, predict, and improve those processes. We define a (MF) as a set of related metrics, data collection mechanisms , and data uses inside a software organization. It is not uncommon to find measurement frameworks that are: (1) collecting redundant data, (2) collecting data that nobody uses, or (3) collecting data that might be useful to people who do not even know the data exist inside their organization. For these reasons , improving ongoing measurement is an important problem for many software organizations. We believe the solution for this problem needs to address two key issues: (1) how to better understand and structure this ongoing measurement, and (2) how to better explore the data that the organization has already collected. This paper describes an approach that addresses these two critical issues jointly. The approach combines a knowledge discovery technique, called attribute focusing (AF), with a measurement planning approach, called the goal-question-metric (GQM) paradigm. In this approach, a GQM-based method is used to understand and structure ongoing measurement , and an …